The Computational Complexity of High-Dimensional Correlation Search

There is a growing awareness that the popular support metric (often used to guide search in market-basket analysis) is not appropriate for use in every association mining application. Support measures only the frequency of co-occurrence of a set of events when determining which patterns to report back to the user. It incorporates no rigorous statistical notion of surprise or interest, and many of the patterns deemed interesting by the support metric are uninteresting to the user. However, a positive aspect of support is that search using support is very efficient. The question we address in this paper is: can we retain this efficiency if we move beyond support, and to other, more rigorous metrics? We consider the computational implications of incorporating simple expectation into the data mining task. It turns out that many variations on the problem which incorporate more rigorous tests of dependence (or independence) result in NP-hard problem definitions.